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ABSTRACT 

Telepresence is an approach to teleoperation that provides egocentric, intuitive interactions between 
an operator and a remote environment. This approach takes advantage of the natural cognitive and 
sensory-motor skills of an on-board crew and effectively transfers them to a slave robot. A dual-arm 
dexterous robot operating under telepresence control has been developed and initial evaluations of the 
system performing candidate EVA, IV A and planetary geological tasks were conducted. The results of 
our evaluation showed that telepresence control is very effective in transferring the operator’s skills to 
the slave robot. However, the results also showed that, due to the kinematic and 

dynamicsinconsistencies between the operator and the robot, a limited amount of intelligent automation 
is also required to carry out some of the tasks. Therefore, several enhancements have been made to the 
original system to increase the automated capabilities of the control system without losing the benefits 
of telepresence. 
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INTRODUCTION 

The current baseline approaches to robot 
teleoperation in the Space Shuttle as well as on 
the International Space Station Alpha (ISSA) 
are based on "joystick" type hand controllers. 
The visual feedback is provided by multiple 
cameras, most of which are mounted on the 
robot arms and at the worksite. For demanding 
tasks that require a high degree of 
coordination, the "joystick" approach is 
inadequate, and may overload the visual and 
manual capacities of the operator. As a result, 
the operator's skill is not effectively 
transferred to the slave robot. A different 
approach to robot teleoperation is telepresence. 
In telepresence, the master control and feedback 
devices are designed to maximize the use of the 
operator's innate cognitive and sensory-motor 
skills [1][3], 

This paper describes an evolving telerobotics 
testbed at the NASA Johnson Space Center 
(JSC) that utilizes virtual reality (VR) and 
telepresence as its baseline mode of operation. 


The testbed consists of a master and a slave 
system. The slave system is a dual-arm 
dexterous robot called the Dexterous 
Anthropomorphic Robotic Testbed (DART). 
DART is controlled by the Full Immersion 
Telepresence Testbed (FITT), which is the 
master system of the overall testbed. FITT 
consists of several VR related input and output 
devices including a speech recognition 
system[3][4]. 

Besides describing the overall system, this 
paper will also discuss the results of our 
preliminary evaluations and the enhancements 
made to improve the capability of the original 
system. 


OBJECTIVES 

The main objective of the DART/FITT testbed 
is to develop and demonstrate technologies 
leading to a highly versatile and productive 
space telerobot. Specific objectives include: (1) 
develop a control scheme that permits the 
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human operator to easily coordinate complex 
robot motions in demanding space tasks; (2) 
improve versatility and productivity; (3) 
maintain compatibility with existing and 
future crew interfaces (e.g., handles, tools); and 
(4) build in the capability for the testbed 
system to evolve. 


DESIGN APPROACHES 

Due to the inability of today's autonomous 
robots to perform complex unplanned tasks in a 
non-stationary, unstructured environment, we 
chose teleoperation as the baseline mode of 
operation. In teleoperation, the human 
operator can provide the cognitive and sensory- 
motor skills necessary to carry out these tasks. 

With teleoperation chosen as the baseline 
mode of operation, the challenge now becomes 
how teleoperation can be made more effective. 
To meet this challenge, we applied the 
telepresence and VR technologies. 

To complement an ergonomic telepresence/VR 
interface, we designed the slave robot to take 
on a human-like configuration and dexterity, so 
that the master-to-slave mapping is more 
direct. 

Finally, we developed an open control 
architecture for shared control and "plug-and- 
play" capability. Shared control can increase 
the robot's productivity through the use of 
automation without sacrificing the versatility 
offered by the human operator. The "plug-and- 
play" modularity of the architecture will 


permit the robot to evolve by incorporating new 
automation capabilities as they emerge. 

Following the above approaches, we developed 
the DART /Fill testbed system for laboratory 
evaluation (Figure 1). The following section 
describes the DART/FITT system in greater 
detail. 


FITT 

The FITT testbed, shown in Figure 2, is centered 
around a motorized chair and includes 
equipment for controlling DART's head camera 
unit, robotic arms and hands. The FITT also 
includes foot pedals that command direct drive 
motors on both the FITT base and the DART 
base, as well as initiate and terminate voice 
commands. 

A VR helmet displays the remote stereo 
camera images with a 60 degree field of view 
and includes stereo head phones for audio 
feedback and a microphone for voice commands. 
The depth perception provided with stereo 
imaging is one of the testbed's most important 
immersion features. A magnetic tracker sensor 
located on the top of the helmet commands the 
orientation of the remote robot's camera head 
unit. The same sensors attached to each of the 
operators wrists provide x,y,z and roll, pitch, 
yaw controls for manipulator tool points. Glove 
controllers worn by the operator read the finger 
joint angles and use the information to control 
the robotic hands. 



Figure 1. Telepresence control of a dual-arm dexterous robot, (a) The Dexterous Anthropomorphic 
Robotic Testbed, (b) The Full Immersion Telepresence Testbed (concept drawing). 
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Since the operator’s hands and eyes are 
virtually immersed in the robot's environment, 
and are not available for initiating keyboard 
commands, the voice recognition system 
provides a convenient means of blending 
automated commands with the baseline 
telepresence control. The operator simply 
presses a foot pedal, gives a verbal command, 
and then releases the pedal. This technique 
prevents the voice recognition system from 
picking up extraneous inputs. The command is 
also processed and played back to the operator 
over a voice synthesizer for confirmation. 
These voice commands vary in complexity from 
a simple repositioning of a robot arm relative to 
a human arm ( to take advantage of the greater 
travel of the robot arm) to a more complex 
maneuver such as grappling onto a dial and 
turning it a preprogrammed number of times. 

The software that communicates and controls 
the FITT systems is hosted on a UNIX/VME 
workstation and a 486 PC equipped with a 
voice recognition board. Data from the 
magnetic tracker sensors, the glove controllers, 
and foot pedals, is sampled at approximately 
100 Hz and then sent out over Ethernet using the 
TeleRobotics Interconnect Protocol (TelRIP)[7]. 



Figure 2. Full Immersion Telepresence Testbed. 


TelRIP is a high-level, object-oriented 
communication package that makes the low 
level socket interfaces transparent to the 
programmer. For example, the voice 
recognition system samples data at a natural 
speaking speed and sends out TelRIP objects 


that initiate a prescribed semi-automated or 
automated action. A remote robot such as 
DART can set up its client communications 
program to receive any or all of the commands 
from FITT by registering "interest" in the 
appropriate TelRIP objects. 

The force-reflective Exoskeleton Arm Master 
(EAM), worn by the operator in Figure 2, is not 
currently integrated with the FITT system. 
Nevertheless, we expect that the operator will 
be able to perform more complex tasks with an 
increased level of performance once the EAM is 
integrated with the FITT system. 


DART 

DART, shown in Figure 3, includes several 
robotic devices, controllers, and supporting 
workstations. The robotic arms are PUMA 
562's each with an 8.8 pound payload 
capability. Each arm also has a force-torque 
sensor. On the right arm is a Stanford /JPL 
hand. Each finger has a urethane fingertip to 
provide a high static friction surface and can be 
hyper-extended to provide a large 
manipulation envelope. On the left arm is a 
parallel jaw gripper. The head camera unit 
that provides video feedback to the 
teleoperator supports 3 axes of rotations and 
contains two color CCD cameras. The driver 
level software is executed on two Tadpole™ 
multiprocessor systems. Each multiprocessor 
system has four M88000 processors and runs a 
multiprocessor version of the UNIX operating 
system. The vision system is implemented on a 
DataCube™ pipeline image processor board. 



Figure 3. The Dexterous Anthropomorphic 
Robotic Testbed PART). 
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CONTROL ARCHITECTURE 

DART and FTTT each has a distributed control 
architecture. Each subsystem spans one or more 
processes. The subsystem processes are 
distributed across several different computers, 
networked on an Ethernet backbone. Figure 4 
shows how the DART and FITT systems are 
networked. Computers to the left of the 
SPARC-10 are part of the DART testbed; 
computers to the right of the SPARC-10 are 
part of the FITT testbed. The SPARC-10 itself 
serves as a message router and hosts the speech 
synthesis software. 

The software architecture is shown in Figure 5. 
The subsystem processes communicate and are 
synchronized by TelRIP. This architecture 
provides a flexible environment for 
development, maintenance, and future 
enhancements. FITT controls DART by linking 
to this Ethernet backbone and commanding the 
subsystems through TelRIP. The router process, 
denoted by R, is responsible for transmitting 
data to the appropriate subsystem processes. 

PRELIMINARY EVALUATIONS 

Preliminary evaluations of the DART/FITT 
system were conducted using operators of 
varying skill levels, ranging from several years 
of robotic experience to absolutely no 
engineering experience. This allows the 
intuitiveness of operation to be qualitatively 


evaluated. The tasks ranged from inspection to 
object handling to dexterous manipulation. 

Inspection tasks were comprised mainly of 
bringing an object towards the head camera and 
viewing it from different angles. These tasks 
provide information about the required display 
resolution, stereo perception, as well as the 
effect of working with egocentric views of the 
workspace. The object handling tasks include 
picking up objects of various sizes and shapes 
(e.g., balls, pipes, tools) and placing them at a 
different location, and handing objects back and 
forth between the dexterous hand and the 
gripper. Some of the dual-hand dexterous tasks 
performed were tying a knot with a rope, 
folding and unfolding a thermal blanket, and 
manipulating an electronic task panel which 
contains toggle and rocker switches, push 
buttons, sliders, and a dial. These tasks reflect 
some of the basic dexterity and skills required 
for on-orbit extra- and intra-vehicular 
activities (EVA/IVA). 

To further evaluate the DART/FITT system as 
a "planetary geologist", we put the system 
through a battary of tasks including holding up 
a light source while the operator examined a 
rock sample, picking up a rock sample and 
placing it into a bag or container, chipping a 
boulder with a hammer, picking up rock 
samples with an extended tong, and placing a 
gnomen next to a rock sample as a scale and 
color reference. 
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Figure 4. The DART/FITT computer configuration. 
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Figure 5. The distributed control architecture of DART/FITT. 


OBSERVATIONS 

One of the most significant observations from 
the preliminary evaluations is the short time 
it takes a new operator to become proficient 
with the system. For example, operators with 
no previous experience were able to transfer 
objects between the two hands and manipulate 
the controls on the panel within a 30 minute 
session. Operators with considerable 
experience in "cock-pit" type control have also 
found the training time greatly reduced due to 
the intuitiveness of the motion controls and the 
immersiveness of the visual feedback. 

We have also identified several areas needing 
improvements. The weight of the exoskeleton 
glove controller caused muscle fatigue when it 
was necessary to maintain a specific position 
for a long period of time. This observation 
suggests the need for a mechanism that will 
allow the operator to re-adjust his or her arm 
positions (e.g., indexing), and to use light- 
weight glove controllers. 

While teleoperation of the dexterous hand 
offers much flexibility for grasping, it was 
found inadequate for manipulation. The 
difficulty lies in the kinematics dissimilarity 
between the robot's and the operator's hands. 

The operator can also experience mild motion 
sickness when using the system due to a slight 
delay between the motions of the operator's 
head and the DART camera system. This only 


occurs when the operator makes large, quick 
head movements. Motion sickness usually 
occurs whenever there is a significant mismatch 
between the robot's and the operator's rate of 
motion. Motion sickness can also be caused by 
unintended body and head movements. 
However, since the operator rarely has to make 
large head movements once focused on a task, 
this problem is not a major prohibiting factor. 

Although the current system provides the 
necessary visual cues to perform many tasks, a 
few limitations of the visual feedback have 
been observed. The visual feedback the 
operator receives is coarse (495 X 240 pixels) 
and the distance between the head cameras is a 
little too narrow, so the depth perception of the 
operator is not optimal. These visual 
limitations can have serious impacts on the 
operator's performance. For example, since 
FITT currently does not offer force-reflection, 
the operator assesses the force imparted onto 
the environment by watching for the amount of 
physical compliance. The active compliance of 
the DART's fingers is very useful in this regard. 

Another problematic area encountered is the 
transformation of human hand motions to 
DART's hand motions. Several transformation 
methods were explored[6]. These methods 
included joint-to-joint mapping, forward and 
inverse kinematics transformations, and a 
combination of joint and Cartesian control. The 
two major difficulties encountered when 
applying these techniques are the dissimilar 
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kinematics of the human’s and DART's hands, 
and the slight changes in the sensor positions 
when the gloves are taken off and put back on. 
Joint-to-joint mapping was chosen as the 
method of control due to the computational 
simplicity and the intuitiveness of the control. 

The telepresence evaluations also revealed 
some interesting operator behaviors. For 
example, an initial exercise is desirable before 
each session to familiarize the operator with 
the system's behavior. The exercise typically 
involves having the operator command the 
robot's arms, hands and head in various 
different ways to explore the dexterity of the 
robot. Without the exercise, less experienced 
operators often have the tendency to move like 
a robot, not fully utilizing his or her natural 
coordination skill. After a few training 
sessions, the operator generally will learn to 
compensate for any kinematics dissimiliarities 
between the operator and the robot. 

Perhaps the most interesting observation was 
that the operator's dependency on the visual 
feedback decreases as a function of the amount 
of training time. This is most evident when the 
operator flipped on/off an electrical switch 
without actually seeing the fingertip making 
contact with the switch. This observation can 
probably be explained by the circular learning 
theory introduced by Piaget[5], and Held and 
Hein[2]. 

Even with the observed limitations, the 
original DART/FITT system was able to 
complete all of the assigned tasks in a 
reasonable amount of time. 


SYSTEM MODIFICATIONS 

After our initial evaluations of the 
DART/FITT system, we began to focus on how to 
overcome the limitations of telepresence 
without losing its benefits. As DART/FITT 
evolves, several features have been added to 
maximize the system's usefulness. These 
features are discussed below. 

To expand the robot's capabilities beyond those 
of the operator's arms and hands, several 
features have been added. First, different 
voice-invoked hand grasp primitives (e.g. 
pinch, cylindrical, hook, etc.) were made 
available to the operator to compensate for the 
kinematic dissimilarities between the human 


hand and the robot hand. We have also 
replaced the exoskeleton glove controllers with 
the light weight CyberGloves™ to reduce 
fatigue. 

Similarly, a ''freeze" voice command was 
added to the system to enable and disable the 
tracking between robot and operator, allowing 
the operator to rest her arms. A "re-index" 
voice command was also added to allow the 
operator to control the robot in a more 
comfortable position. The "re-index" command 
also allows the operator to fully utilize the 
joint and reach capabilities of the robotic arms. 

In a shared control scheme where the operator 
and the automated control primitives both 
have access to the robot, a method must be 
provided to coordinate their interactions. In 
the case of FITT, a speech recognition system 
was selected as a "hands-free" method for the 
operator to communicate to the robot. The 
operator issues commands through a 
microphone located on the FITT helmet. For 
example, the operator would say "spherical 
grasp" to change the configuration of the hand, 
or "freeze left arm” to disable tracking between 
the operator’s left arm and the robot's left arm. 

To expand viewing capabilities, a wrist camera 
was added to DART's right arm. The operator 
can switch from the "head view” to the "wrist 
view” for aligning the hands when grasping 
visually obstructed objects. Also an advanced 
pipeline-based vision system is being created to 
perform shape recognition, target location, and 
closed-loop visual servoing of robotic arms for 
grasping. 

The mild motion sickness experienced by the 
operator when DART was rotating has been 
relieved by having the operator platform (a 
motorized chair) rotate along with the DART 
base. The acceleration and deceleration cues 
provide the operator with sufficient 
kinesthetic feedback to prevent disorientation. 


FUTURE WORK 

The early evaluations have demonstrated the 
versatility of the DART/FITT system and 
confirmed the feasibility of our approach. 
However, to further improve the system's 
productivity, the intelligent automation 
aspects of the system must be expanded. For 
example, the hand will be able to manipulate 
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latches and handles on space hardware, as 
well as flexible objects such as plastic sample 
bags through automated sequences. Automated 
arm modes will be incorporated for tasks such 
as using a hammer to chip a rock for planetary 
exploration. 

Several arm upgrades are planned. A control 
system with coordinated motion between the 
two arms and hands will be added to the 
system for use with dual arm grasps. An 
additional capability, scaling, where the ratio 
of the amount the robot moves to the amount 
the operator moves can be changed, will be 
added to the system in order to make fine 
motion control of the arms easier. 

The advanced vision system will be enhanced 
to provide basic perception of the environment 
needed for automated manipulation and 
grasping. Such capability will be especially 
important for planetary applications since the 
communication delay between the operator and 
robot may be large. 

A second generation head camera unit will be 
fabricated to provide tighter head tracking 
and to correct the narrow interpupilary 
distance. A high-resolution (640 X 480 pixels) 
head-mounted displays will be sought to 
improve the operator's visual acuity. 

The force-reflective dexterous arm master, 
(Figure 2) will be integrated with FITT to 
evaluate the effects of force-reflection. 
Additional evaluations will be conducted to 
quantify the performance of the DART/FITT 
system. New test subjects will be recruited to 
study the correlation between training-time 
versus performance, and the performance of 
"cock-pit" type control versus telepresence. 

Virtual reality simulation of the robot will be 
developed and over-laid into the VR helmet as 
a predictive display. Virtual instrument 
displays such as bar graphs and meters will be 
used to assist the operator in various tasks. 

CONCLUSIONS 

Telepresence is not a new idea. It is, however, 
an idea that is becoming a reality due to the 
recent advances in head-mounted displays, 
dexterous glove controllers, motion trackers, 
force-reflective masters, and other human 
compatible interactive devices. The DART and 


FITT combination represents an integration of 
these telepresence and VR technologies for 
space robotics applications. While further 
evaluations will be necessary to completely 
characterize the system, we believe all of our 
stated objectives have been met. Many lessons 
were learned in our preliminary evaluations 
and several areas for improvement were 
identified. Our future work will address these 
areas. However, the benefit of telepresence 
and VR in space robotics is clearly evident by 
the variety of complex tasks DART/FITT can 
perform under the control of an operator. 
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